Goto

Collaborating Authors

 audio-visual spatial alignment


Review for NeurIPS paper: Learning Representations from Audio-Visual Spatial Alignment

Neural Information Processing Systems

Saying the models completely disregard spatial information is too strong a statement as these models can easily be repurposed to localize sound sources to some extent. I believe there is some miscommunication. I meant using the model for a downstream task that requires audio visual spatial alignment. The authors report results of the AVSA self-supervision task and compare it to other methods like AVC. But that is the self-supervision task or pre-text task setup rather than an actual downstream task.


Learning Representations from Audio-Visual Spatial Alignment

Neural Information Processing Systems

We introduce a novel self-supervised pretext task for learning representations from audio-visual content. Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances. Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time. While these approaches learn high-quality representations for downstream tasks such as action recognition, they completely disregard the spatial cues of audio and visual signals naturally occurring in the real world. To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360\degree video and spatial audio.